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(54) Title: HISTOLOGICAL RECONSTRUCTION AND AUTOMATED IMAGE ANALYSIS 
(57) Abstract 



A method for automated image analysis (Fig. 
2, 42) of a biological specimen by histological recon- 
struction, provided is an automated cell image method 
for analyzing a biological specimen that has consec- 
utively been stained by either an in situ hybridiza- 
tion (ISH) method, or an immunohistochemistry (IHC) 
method or a nucleic acid stain, and counterstained. 
The method couples composite images in an automated 
manner for processing and analysis to identify struc- 
ture in tissue that cannot be captured in a single field 
of view image (Fig. 2, 44) or a single staining tech- 
nique, the disclosure provides a method for histolog- 
ical reconstruction to analyze many fields of view on 
potentially many slides (Fig. 2, 37) simultaneously. 
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WO 00/62247 PCT/US00/09479 
HISTOLOGICAL RECONSTRUCTION AND AUTOMATED IMAGE ANALYSIS 

TECHNICAL FIELD 
The invention relates generally to light microscopy 
and, more particularly, to automated techniques of 
5 analyzing cytochemical and immunohistochemical staining. 



BACKGROUND OF THE INVENTION 

In current cellular imaging systems, the area of a 
stained slide containing a stained cellular specimen is 

10 scanned by automated microscopy systems. The entire 
cellular specimen area of the slide is imaged with a 
series of field of view images. For further analysis, 
each field of view image must be separately analyzed to 
locate complete candidate objects of interest within the 

15 field of view image. This approach may be acceptable for 
cellular objects or clusters that are not so large that 
they extend beyond the field of view width or length of 
an image. Often, for both automated and manual analysis, 
only a single field of view is analyzed for morphological 

20 characteristics, so that the context of the analysis is 
limited to that field of view on a single slide. 



WO 00/62247 PCT/USOO/09479 

2 

SUMMARY OF THE INVENTION 

A method for automated image analysis of a biological 
specimen is disclosed. A biological specimen is stained 
and counterstained for a specific marker or in the 

5 instance of immunohistochemistry or in situ 

hybridization, the marker is detectably labeled. Such 
labels include enzyme, radioisotopes, fluorescence or 
other labels well known in the art. The sample is then 
scanned at a plurality of positions by a photoimaging 

10 system to acquire an image. Adjacent positions are then 
used to reconstruct and provide a full picture image of 
the whole sample. A reconstructed whole sample image may 
then be further processed to identify coordinates that 
may have an object or area of interest. These 

15 coordinates are automatically selected for a candidate 
object of interest. In a preferred embodiment, a low 
magnification image of the candidate objects of interest 
is automatically obtained. Preferably the image is a 
color digital image. The candidate object of interest 

20 pixels in each sample are automatically morphologically 
processed to identify artifact pixels and the remaining 
candidate object of interest pixels in the sample not 
identified as artifact pixels. The apparatus obtaining 
the low magnification image is adjusted to a higher 
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magnification, to acquire a higher magnification image of 
the sample, at the location coordinates corresponding to 
the low magnification image, for each candidate object or 
area of interest. Pixels of the higher magnification 

5 image in the first color space are automatically 

transformed to a second color space to differentiate 
higher magnification candidate objects of interest pixels 
from background pixels and identify, at higher 
magnification, objects of interest from the candidate 

10 object of interest pixels in the second color space. 

Thus, the pathologist or technician can identify whether 
the candidate object of interest has been specifically 
stained for the marker of interest, or counterstained, or 
both specifically stained and counterstained. 

15 An automated cellular image method for analyzing a 

biological specimen, that has consecutively been stained 
with hematoxylin and eosin (H/E) on one tissue section 
and by one or several immunohistochemistry (IHC) and/or 
in situ hybridization (ISH) methods on parallel tissue 

20 sections, is also disclosed as a particular embodiment. 
To identify a structure in tissue that cannot be captured 
in a single field of view image or a single staining 
technique, the invention provides a method for 
histological reconstruction to analyze many fields of 
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view on potentially many slides simultaneously. The 
method couples composite images in an automated manner 
for processing and analysis. A slide on which is mounted 
a biological specimen stained to identify structure of 
5 interest is supported on a motorized stage. An image of 
the biological specimen is generated, digitized, and 
stored. As the viewing field of the objective lens is 
smaller than the entire biological specimen, a 
histological reconstruction is made. These stored images 
10 of the entire tissue section may then be placed together 
in an order such that the H/E stained slide is paired 
with the immunohistochemistry slide so that analysis of 
the images may be performed simultaneously by the 
pathologist. The images may be superimposed or viewed as 
15 adjacent images. 

In one embodiment, the invention provides a method 
for automated image analysis of a biological specimen by 
providing a biological sample to be analyzed, 
automatically scanning the sample at a plurality of 
20 coordinates, automatically obtaining an image at each of 
the coordinates, reconstructing an image of the sample 
from each individual image to create a reconstructed 
image and processing the reconstructed image to identify 
a candidate object or area of interest. 
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In another embodiment, the invention provides a 
method for histological reconstruction. In this method a 
sample of a biological specimen is divided into a number 
of subsamples. Each subsample is processed with a stain, 

5 counters tain, immunohistochemical technique, in situ 

hybridization technique, or a combination thereof. Each 
sample is then scanned and an image is obtained from each 
of the samples. The images are then reconstructed such 
that a first image is paired with a consecutive- 

10 corresponding sample image for identification of objects 
or areas of interest. 

In yet another embodiment, the invention provides a 
computer program residing on a computer-readable medium, 
for automated image analysis of a biological specimen. 

15 The computer program comprises instructions for causing a 
computer to process a sample by scanning the sample at a 
plurality of coordinates, obtaining an image at each of 
the coordinates, reconstructing the sample from the 
individual images to create a reconstructed sample, 

20 identifying a coordinate of a candidate object or area of 
interest in the reconstructed image and acquiring an 
image at the coordinate obtained from the reconstructed 
image . 
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DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a perspective view of an apparatus for 
automated cell image analysis. 

FIG. 2 is a block diagram of the apparatus shown in 
5 FIG. 1. 

FIG. 3 is a plan view of the apparatus of FIG. 1 
having the housing removed. 

FIG. 4 is a side view of a microscope subsystem of 
the apparatus of FIG. 1. 
10 FIG. 5 shows a slide carrier. FIG. 5a is a top view 

of a slide carrier for use levity the apparatus of FIG. 
1. FIG . 5b is a bottom view of the slide carrier of FIG. 
5a. 

FIG. 6 shows views of an automated slide handling 
15 subsystem. FIG. 6a is a top view of an automated slide 
handling subsystem of the apparatus of FIG. 1. FIG. 6b 
is a partial cross-sectional view of the automated slide 
handling subsystem of FIG. 6a taken on line A-A. 

FIG. 7a-7d illustrate the output operation of the 
20 automated slide handling subsystem. 

FIG. 8 is a flow diagram of the procedure for 
automatically determining a scan area. 

FIG. 9 is a block diagram of the microscope 
controller of FIG. 2 
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FIG. 10 shows a method of histological 
reconstruction. 
DETAILED DESCRIPTION 

Overview 

5 An automated microscope for analyzing cellular 

specimens is shown in FIG's. 1 and 3 and in block diagram 
in FIG. 2. A motorized stage 38 may be used to support a 
slide 70 (FIG. 5) . On the slide is mounted a biological 
specimen that is typically stained to identify a 

10 structure of interest for analysis. A biological 

specimen is defined as a cellular or acellular sample of 
biological origin, which has been mounted on a microscope 
slide. The biological specimen can be, for example, a 
tissue section, a biological fluid specimen, for example, 

15 a blood fraction cytospun on a microscope slide or a cell 
suspension directly seeded on a slide. 

At least one optical sensing array, such as an 
objective lens, 44a is located above the stage and a 
light source 4 8 is located beneath the stage. Light from 

20 the source illuminates the stage and slide so an image of 
the biological specimen is generated by the optical 
sensing array, for example an objective lens. This image 
is stored in memory. Preferably the image is a digitized 
or digital image. As the viewing field of the optical 
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array is smaller than the entire biological specimen, the 
stage is moved in one planar direction by a distance that 
corresponds to the length of the field of view in that 
direction. The image generated at that position may then 

5 be captured and stored. The acquired image may be 

flipped along its centerline due to the optical flipping 
of the original image. Movement of the stage and capture 
of the resulting image continues in the same direction 
until the end of the specimen area of the slide is 

10 reached. At that time, the stage is moved in the other 
planar direction by a distance that corresponds to the 
length of the field of view in that direction and another 
image is generated and stored. The slide is traversed or 
scanned in this manner until the entire specimen area of 

15 the slide has been viewed through the objective lens. 
These stored images may then be placed together in the 
order in which they were collected to generate a 
composite or reconstructed image of the cellular 
specimen. This composite image may then be analyzed to 

20 detect a structure that extends across more than one 

image field or more than one slide for further analysis. 
Such analysis may result in the identification of a 
candidate object or area of interest in both the field of 
view as well as objects that overlap two or more fields 
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of view. In such instances, the system will 
automatically determine the coordinates for these 
candidate objects and may obtain additional images at 
various magnifications. 
5 In existing cellular imaging systems, the area of the 

slide is scanned by automated microscopy systems to image 
the entire specimen area of the slide with a series of 
field of view images in the manner described above. 
However, these systems require that each field of view 
10 image is separately analyzed to locate complete candidate 
objects of interest within the field of view image for 
further analysis. This approach is acceptable for 
cellular objects that are not so large that they extend 
beyond the field of view width or length of an image. To 
15 identify structure, in tissue that cannot be captured in a 
single field of view image, the present system uses an 
analysis method to identify field of view images that 
appear to contain part of a tissue structure to be 
analyzed. Field of view images so identified that are 
20 adjacent to one another are then identified as containing 
the tissue structure that the stain, antibody, or probe 
was intended to identify. This portion of the composite 
image may then be viewed under a higher magnification 
power for additional detail. The automatic building of a 



WO 00/62247 rt PCT/US00/09479 

10 

composite specimen image from the field of view images 
and the processing of the composite image or its 
constituent images to identify tissue structure that 
extends beyond one field of view is unknown. 

5 In addition, a problem with existing automated 

systems is the continued need for operator input to 
initially locate cell objects for analysis. Such 
continued dependence on manual input can lead to errors 
including objects of interest being missed. These errors 

10 can be critical especially in assays for so-called rare 
events, e.g., finding one stained cell in a cell 
population of one million normal cells. Additionally, 
manual methods can be extremely time consuming and can 
require a high degree of training to properly identify or 

15 quantify cells. The associated manual labor leads to a 
high cost for these procedures in addition to the 
potential errors that can arise from long, tedious manual 
examinations. A need exists, therefore, for an improved 
automated cell image analysis system which can quickly 

20 and accurately scan large amounts of biological material 
on a slide. 

Unless otherwise defined, all technical and 
scientific terms used herein have the same meaning as 
commonly understood by one of ordinary skill in the art 
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to which this invention belongs . Although methods and 
materials similar or equivalent to those described herein 
can be used in the practice or testing of the present 
invention, suitable methods and materials are described 
5 below. 

Nuclear Stains, Intercalating Dyes and Counters tains 
The term "nuclear stain" refers to a cytochemical 
stain that preferentially stains the nuclei of eukaryotic 
cells. Many nuclear stains are intercalating dyes. The 

10 term "intercalating dye" refers to a chemical compound 

that can insert itself in between adjacent nucleotides of 
a nucleic acid to provide a detectable color. 

Many nuclear stains are known in the art, with one of 
the most commonly used being hematoxylin. Hematoxylin is 

15 often used in combination with various metallic salts 
(mordants) . Hematoxylin stains are used for different 
staining purposes, and have a variety of colors, 
depending on the metal used. Aluminum lakes are purple 
to blue, depending on pH. Iron lakes are blue-black. 

20 Chromium lakes are blue-black. Copper lakes are blue- 
green to purple. Nickel lakes are various shades of 
violet. Tin lakes are red. Lead lakes are dark brown. 
Osmium lakes are greenish brown. Other nuclear stains 
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include Giemsa stain, methyl green (which binds to AT- 
rich DNA regions), and Nuclear Fast-Red. 

Fluorescent stains include Hoechst 33342; Hoechst 
33258 (Calbiochem) , a bisbenzimide DNA intercalator that 
5 excites in the near UV (350 nm) and emits in the blue 

region {450 nm) ; thiazole orange, a fluorogenic stain for 
DNA that excites in the blue region (515 nm) and emits in 
the green region (530 nm) of the visible spectrum; DAP I; 
ethidium bromide; propidium iodide; TOTO; YOYO__l; and 
10 SYTOX Blue or Green stains are also encompassed by the 
current invention. Several dyes either bind GC-rich or 
AT-rich chromosomal regions preferentially or show 
differences in fluorescence intensity upon binding those 
regions, yielding fluorescent banding patterns. For 
15 example, 7-Aminoactinomycin D binds selectively to GC- 
rich DNA regions and. 9-Amino-6-chloro-2-methoxyacridine 
fluoresces with greatest intensity in AT-rich DNA 
regions. Acridine homodimer fluoresces preferentially 
when bound to AT-rich DNA regions . 
20 The term "counterstain, " when used in combination 

with nuclear stains, refers to cytochemical stains that 
bind to a region of a eukaryotic cell other than the 
nucleus. Many counterstains are known in the art. One of 
the most common is eosin, which stains eukaryotic cell 
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cytoplasm to varying shades of pink. Other counterstains 
are specific for a particular organelle or a protein in a 
cell. For example, the Kleihauer-Betke cytochemical stain 
is specific for hemoglobin F, a hemoglobin type 
5 preferentially expressed in fetal cells and therefore can 
be defined as a specific marker of fetal red blood cells. 
The term "coordinate" or "address" is used to mean a 
particular location on a slide or sample. The coordinate 
or address can be identified by any number of means 
10 including, for example, X-Y coordinates, r-P coordinates, 
and others recognized by those skilled in the art. 

In one embodiment, an automated cellular imaging 
method is used to identify fetal nucleated red blood 
cells in a maternal blood specimen. Fetal cells are first 
15 identified following the Kleihauer-Betke cytochemical 

stain for hemoglobin F. Fetal cells are identified by the 
automated cellular imaging system as objects on the basis 
of their bright red color (indicative of Hemoglobin F) as 
compared to maternal red blood cells. To assure that 
20 appropriate objects are identified, size and shape 

morphological "filters" are used to exclude very small 
and very large objects. 

Cells are counterstained with an additional 
cytochemical stain for nucleic acids, resulting in a blue 
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color for nucleated red blood cells (generally, fetal red 
blood cells) . An automated image analysis system 
identifies blue objects of the appropriate size and shape 
for an erythrocyte nucleus among the bright red objects, 
5 allowing the imaging system to identify and enumerate 
nucleated fetal red cells. Such cells can be enumerated, 
allowing for a screen for Down's syndrome in the fetus, 
wherein the frequency of such cells is typically higher 
in Down's syndrome pregnancies compared with normal 
10 pregnancies. 

In a preferred embodiment, the slides are stained 
with hematoxylin/eosin (H/E) and one or several parallel 
slides containing adjacent sections are stained for one 
or several specific markers. 
15 The results of the H/E staining provide cells with 

nuclei stained blue-black, cytoplasm stained varying 
shades of pink; muscle fibers stained deep pinky red; 
fibrin stained deep pink; and red blood cells stained 
orange-red. 

20 For example, hematoxylin/eosin (H/E) slides are 

prepared with a standard H/E protocol. Standard 
solutions include the following: (1) Gills hematoxylin 
(hematoxylin 6.0 g; aluminum sulphate 4.2 g; citric acid 
1,4 g; sodium iodate 0.6 g; ethylene glycol 269 ml; 
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distilled water 680 ml); (2) eosin (eosin yellowish 1.0 
g; distilled water 100 ml); (3} lithium carbonate 1% 
(lithium carbonate 1 g; distilled water 100 g) ; (4) acid 
alcohol 1% 70% (alcohol 99 ml cone; hydrochloric acid 1 
5 ml); and (5) Scott's tap water. In a beaker containing 1 
L distilled water, add 20 g sodium bicarbonate and 3.5 g 
magnesium sulphate. Add a magnetic stirrer and mix 
thoroughly to dissolve the salts. Using a filter funnel, 
pour the solution into a labeled bottle. 
10 The staining procedure is as follows: (1) Bring the 

tissue or cell sections to water; (2) place sections in 
hematoxylin for 5 minutes (min) ; (3) wash in tap water; 
(4) 'blue' the sections in lithium carbonate or Scott's 
tap water; (5) wash in tap water; (6} place sections in 
15 1% acid alcohol for a few seconds; (7) wash in tap water; 
(8) place sections in eosin for 5 min; (9) wash in tap 
water; and (10) dehydrate with graded alcohol solution. 
Mount sections . 

A specific marker is a molecule or a group of 
20 molecules, which is present in only a subset of the 
components of a biological specimen and therefore 
identifying specifically the components having the 
marker. Specific markers are frequently defined as 
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antigens recognized by specific antibodies (monoclonals 
or polyclonals) and can be detected by 
immunohistochemistry. 

Another group of specific markers is defined by the 
5 capacity of these markers to hybridize, specifically, a 
nucleic acid probe. These markers can usually be 
detected by in situ hybridization. 

A third group of specific markers can be defined by 
their enzymatic activity and can be detected by 
10 histochemistry. 

A fourth group of specific markers can be stained 
directly, histochemically, using a specific dye. 

A fifth group of specific markers can be defined as 
being receptors binding specifically to one or several 
15 ligands. A specific ligand is itself used for the 
detection of the receptor-ligand complex, using a 
detection method involving either histochemistry, or 
immunohistochemistry or in situ hybridization. 

Immunohistochemical and In Situ Hybridization 
20 Techniques 

Immunohistochemical techniques as used herein 
encompasses the use of reagents detecting cell specific 
markers, such reagents include, for example, as 
antibodies and nucleic acids probes. Antibodies, 
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including monoclonal antibodies, polyclonal antibodies 
and fragments thereof, are often used to identify 
proteins or polypeptides of interest in a sample. A 
number of techniques are utilized to label objects of 

5 interest according to immunohistochemical techniques . 
Such techniques are discussed in Current Protocols in 
Molecular Biology, Unit 14 et seq., eds . Ausubel, et al . , 
John Wiley & Sons, 1995, the disclosure of which is 
incorporated herein by reference. For example, the 

10 following procedure is an example of immunohistochemical 
staining using an antibody recognizing, specifically, the 
HER2 protein. HER2 overexpression has been described as 
a specific marker in a high percentage of breast cancer 
carcinomas . This protocol is intended for staining a 

15 paraffin embedded tissue section prepared according to a 
conventional procedure. 

The section is deparaf f inized using two baths of 
xylene and rehydrated through graded alcohols baths and 
finally in deionized water. The section is then 

20 incubated with an Antigen Retrieval Buffer, containing 
Citrate, for 4 0 minutes at 95 °C. The slide is then 
cooled down at room temperature for 20 minutes in the 
same buffer. The slide is then rinsed in deionized 
water. 
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The area surrounding the tissue section is carefully 
dried and a hydrophobic delimiting pen is used to draw a 
line around the specimen, on the glass slide. A 
peroxidase blocking solution is added on the section and 

5 incubated 5 minutes at room temperature. After being 
washed twice with the Wash Buffer {a balanced salt 
solution) , the tissue section is incubated 30 minutes at 
room temperature, with the primary antibody recognizing 
the HER2 protein. 

10 After 3 washes with the Wash Buffer, the tissue 

section is incubated with the peroxidase conjugated 
secondary antibody. The secondary antibody will 
recognize specifically the primary antibody. The slide 
is then washed 3 times with the Wash Buffer. Then the 

15 tissue section is incubated in presence of DAB and 

hydrogen peroxide for 10 minutes, before being washed 
with water. 

The tissue section is counterstained in Hematoxylin 
for 2 minutes and rinsed again with water. The slide is 
20 mounted with a cover-slip using an aqueous mounting 
medium. 

Immunohistochemical localization of cellular 
molecules uses the ability of antibodies to bind specific 
antigens, for example proteins of interest such as onco- 
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proteins and enzymes, with high affinity. These 
antibodies can be used to localize antigens to 
subcellular compartments or individual cells within a 
tissue. 

5 In situ hybridization techniques include the use of 

specifically labeled nucleic acid probes, which bind to 
cellular RNA or DNA in individual cells or tissue 
section. Suitable nucleic acid probes may be prepared 
using standard molecular biology techniques including 

10 subcloning, plasmid preparation, and radiolabeling or 
non-radioactive labeling of the nucleic acid probe. 

In situ hybridization is often performed on either 
paraffin or frozen sections. Such techniques often 
include fine sectioning of tissues to provide samples 

15 that are only a single to a few cell layers thick. For 
example paraffin blocks containing a tissue sample are 
cut into thin, approximately 8 um tissue sections, which 
are subsequently mounted on subbed slides to be further 
processed for in situ hybridization. Alternatively, 

20 methacrylate may be used for sectioning. Cryosectioning 
techniques are particularly suitable for 
immunohistochemistry and enzyme histochemistry. 

Immunofluorescent labeling of a tissue section often 
use a sandwich assay or a primary antibody and secondary 
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antibody- fluorochrome conjugate. Slides containing a 
tissue section of interest are washed in phosphate 
buffered saline and then exposed to a primary antibody 
which will bind to the protein object of interest. 
5 Subsequently the slides are washed and exposed to the 
secondary antibody which binds to the first or primary 
antibody. The slide is washed and then developed. 
Numerous other techniques well known in the art of 
immunohistochemical staining and in situ hybridization 
10 are easily adaptable for use in immunohistochemical 
reconstruction as disclosed herein. 

Thus, a combination of techniques using both chemical 
staining and/or immunohistochemical and/or in situ 
hybridization may be used in the present methods. For 
15 example, numerous subsamples may be prepared from a 
single tissue specimen. A first subsample may be 
chemically stained as discussed above, and a subsequent 
subsample may be subjected to immunohistochemical and in 
situ hybridization techniques. Each subsamples is 
20 scanned and processed as discussed below. Reconstructed 
images may then be overlapped or processed to further 
identify objects of interest. 
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Histological Reconstruction 

Histological reconstruction is a process whereby an 
image of a whole specimen is constructed from analyzed 
pieces of the specimen, particularly when the biological 

5 specimen has been mounted on a slide. This image is 
created by piecing together more than one field of view 
at any particular magnification. 

With reference to Fig. 10, an image 302, representing 
an objective's field of view is acquired at a first 

10 particular coordinate on the slide sample 301. The slide 
is automatically repositioned on the X-Y stage to obtain 
a new or second field of view corresponding to a second 
particular coordinate 303. This new field of view is 
preferably immediately adjacent to the first field of 

15 view, however, so long as the coordinates, thus the 

address/identity, of each field of view are retained in 
the imaging system, histological reconstruction may be 
performed. This process is repeated until images for the 
whole of the specimen have been acquired. 

20 Based upon each image's X and Y coordinate, the 

specimen is digitally reconstructed. As part of the 
reconstruction, the image may be flipped to correct for 
the optical flipping of the original image. 
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The process of forming a histological reconstructed 
image involves having the apparatus scan a microscope 
slide of interest, and form the image that constitutes a 
reconstruction of the images taken during the scan. The 

5 image that is formed can be a full-color reconstruction 
of the entire scan area, or a fraction of the whole scan 
area, for example, reconstruction of the entire scan area 
that identifies objects or areas of interests. The 
reconstructed digital image can then be used for further 

10 processing or analysis to identify previously undetected 
objects or areas of interest. For example, objects or 
areas of interest overlapping one or more fields of view 
or slides may thus be identified in the reconstructed 
digital image. 

15 With reference to FIGURES 1 and 2, the apparatus 10 

also referred to as the system, comprises a microscope 32 
with a motorized X, Y and Z stage 38, a camera 42, a 
computer 22 adapted to receive and process video images, 
and a set of software programs to control the apparatus 

20 and to execute the method. A measurement of the optical 
properties of the sample features is used to form an 
image of the scannable area of the slide, to find sub- 
regions of interest, and to analyze the properties of 
these regions. The image processing method that 
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evaluates the sample to find regions of interest uses a 
measure of the hue, saturation or intensity and 
luminosity of a 24-bit color image to produce a white on 
black target image of interest. This image is processed 

5 by separately converting the full color image to 
components of hue, saturation or intensity and 
luminosity, thresholding the components, and performing a 
logical "AND" between the two images, then thresholding 
the resulting image such that any pixel value above zero 

10 becomes 255. The processing and image acquisition will 
be further understood with reference to the apparatus 
described below. 



Automated System 

15 The invention provides a method for automated 

analysis of a biological specimen, which eliminates the 
need for operator input to locate biological objects or 
areas of interest for analysis . 

With reference to FIGURE 1, a slide prepared with a 

20 biological specimen and reagent is placed in a slide 

carrier 60 (FIG. 5) which preferably holds four slides. 
The slide carriers are loaded into an input hopper 16 of 
the automated system 10. The operator then enters data 
identifying the instrument protocol which contains 
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information on the size, shape and location of a scan 
area on each slide, or, preferably, the system 
automatically locates a scan area for each slide during 
slide processing. The operator then activates the system 
5 10 for slide processing. At system activation, a slide 
carrier 60 is positioned on an X-Y stage 38 of an optical 
system, such as microscope subsystem 32. Any bar codes 
used to identify slides are read and stored for each 
slide in the carrier. The entire slide is rapidly 
10 scanned at a low magnification, typically lOx. At each 
location of the scan, a low magnification image is 
acquired and processed to detect candidate objects or 
areas of interest. Preferably, color, size and shape are 
used to identify objects or areas of interest. The 
15 location of each candidate object or area of interest may 
be stored by reference to its coordinates or address. 
Each field of view may also be stored as part of a larger 
composite image {described in detail elsewhere) . 

At the completion of the low level scan for each 
20 slide in the carrier on the stage, the optical system may 
be adjusted to a higher magnification such as 40x or 60x, 
for additional specimen processing and image acquisition, 
and the X-Y stage is positioned to the stored locations 
for the candidate objects or areas of interest on each 
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slide in the carrier. A higher magnification image is 
acquired for each candidate object or area of interest 
and a series of image processing steps are performed to 
confirm the analysis, which was performed at low 
5 magnification. A higher magnification image is stored 
for each continued object or area of interest. These 
images are then available for retrieval by a pathologist 
or cytotechnologist to review for final diagnostic 
evaluation. Having stored the location of each object or 
10 area of interest, a mosaic comprising the candidate 

objects or areas of interest for a slide may be generated 
and stored. The pathologist or cytotechnologist may view 
the mosaic or may also directly view the slide at the 
location of an object or area of interest in the mosaic 
15 for further evaluation. The mosaic may be stored on 

magnetic or optical media for future reference or may be 
transmitted to a remote site for review or storage. The 
entire process involved in examining a single slide takes 
on the order of 4-100 min depending on scan area size and 
20 the number of detected candidate objects of interest. 

The invention has utility in prenatal diagnosis of 
fetal cells, rare event detection, rapid cell counting, 
tissue evaluation, and other diagnoses. 
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The processing of images acquired in the automated 
scanning preferably includes the steps of transforming 
the image to a different color space, here preferably by 
hue, saturation and intensity. The pixels of the filtered 
5 image are dynamically thresholded to suppress background 
material; performing a morphological function to remove 
artifacts from the thresholded image; analyzing the 
thresholded image to determine the presence of one or 
more regions of connected pixels having the same color; 
10 and categorizing every region having a size greater than 
a minimum size as a candidate object or area of interest. 

According to another aspect, the scan area is 
automatically determined by scanning the slide; acquiring 
an image at each slide position; analyzing texture or 
15 color information for each image to detect the edges of 
the specimen and storing the locations corresponding to 
the detected edges to define the scan area. 

According to yet another aspect, automated focusing 
of the optical system is achieved by initially 
20 determining a focal surface from an array of points or 
locations in the scan area. The derived focal surface 
enables subsequent rapid automatic focusing in the low 
power scanning operation. In one embodiment, the focal 
plane is determined by determining proper focal positions 
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across an array of locations and performing a least 
squares fit of the array of focal positions to yield a 
focal plane across the array. Preferably, a focal 
position at each location is determined by incrementing 
5 the position of a Z stage for a fixed number of coarse 
and fine iterations . At each iteration, an image is 
acquired and a pixel variance, morphological gradient or 
other optical parameter about a pixel mean for the 
acquired image is calculated to form a set of evaluation 
10 data. The peak value of the least squares fit curve is 
selected as an estimate of the best focal position. 

In another aspect, a focal position method for a 
higher magnification locates a region of interest 
centered about a candidate object of interest within a 
15 slide which was located during an analysis of the low 
magnification images. The region of interest is 
preferably n columns wide, where n is a power of 2. The 
pixels of this region are then processed using a Fast 
Fourier Transform to generate a spectra of component 
20 frequencies and corresponding complex magnitude for each 
frequency component. Preferably, the complex magnitude 
of the frequency components which range from 25% to 75% 
of the maximum frequency component are squared and summed 
to obtain the total power for the region of interest. 
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This process is repeated for other Z positions and the Z 
position corresponding to the maximum total power for the 
region of interest is selected as the best focal 
position. This focal method can be used with many stains 

5 and types of cellular specimens. 

According to still another aspect, a method for 
automated slide handling is provided. A slide is mounted 
onto a slide carrier 60 (FIG. 5) with a number of other 
slides side-by-side. The slide carrier 60 is positioned 

10 in an input feeder 16 with other slide carriers to 

facilitate automatic analysis of a batch of slides. The 
slide carrier is loaded onto the X-Y stage 38 of the 
optical system 32 for the analysis of the slides thereon. 
Subsequently, the first slide carrier is unloaded into an 

15 output feeder 18 after automatic image analysis and the 
next carrier is automatically loaded. 

Referring to the FIGURES, an apparatus for automated 
cell image analysis of biological specimens is generally 
indicated by reference numeral 10 as shown in perspective 

20 view in FIG. 1 and in block diagram form in FIG. 2. The 
apparatus 10 comprises a microscope subsystem 32 housed 
in a housing 12. The housing 12 includes a slide carrier 
input hopper 16 and a slide carrier output hopper 18. A 
door 14 in the housing 12 secures the microscope 
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subsystem from the external environment. A computer 
subsystem comprises a computer 22 having two system 
processors 23, an image processor 25 and a communications 
modem 29. The computer subsystem further includes a 

5 computer monitor 26 and an image monitor 27 and other 
external peripherals including storage device 21, 
pointing device 30, keyboard 28 and color printer 35. An 
external power supply 24 is also shown for powering the 
system. Viewing oculars 20 of the microscope subsystem 

10 project from the housing 12 for operator viewing. The 
apparatus 10 further includes a 3-chip CCD camera 42 for 
acquiring images through the microscope subsystem 32. 
The computer directly controls a number of microscope 
subsystem functions described further in detail. 

15 An automatic slide feed mechanism 37 in conjunction with 
X-Y stage 38 provide automatic slide handling in the 
apparatus 10. An illumination light source 4 8 projects 
light onto the X-Y stage 38 which is subsequently imaged 
through the microscope subsystem 32 and acquired through 

20 the 3-chip CCD camera 42 for processing by the image 

processor 25. A Z stage or focus stage 46 under control 
of the microscope controller 31 provides displacement of 
the microscope subsystem in the Z plane for focusing. 
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The microscope subsystem 32 further includes a motorized 
objective turret 44 for selection of objectives. 

The apparatus 10 is for the unattended automatic 
scanning of prepared microscope slides for the detection 

5 and counting of candidate objects or areas of interest, 
such as stained cells. In one embodiment the invention 
may be utilized for tissue analysis. In another 
embodiment, rare event detection in which there may be 
only one candidate object of interest per several hundred 

10 thousand normal cells, e.g., one to five candidate 

objects of interest per 2 square centimeter area of the 
slide. The apparatus 10 automatically locates and counts 
candidate objects or areas of interest and estimates 
normal cells present in a cellular specimen on the basis, 

15 for example, of color, size and shape characteristics. A 
biological specimen may be prepared with a reagent to 
obtain a colored insoluble precipitate. The apparatus, in 
one embodiment, is used to detect this precipitate as a 
candidate object or area of interest. 

20 During operation of the apparatus 10, a pathologist 

or laboratory technician mounts prepared slides onto 
slide carriers. A slide carrier 60 is illustrated in 
FIG. 5 and is described further below. Each slide 
carrier holds up to 4 slides. Up to 25 slide carriers 
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are then loaded into input hopper 16. The operator can 
specify the size, shape and location of the area to be 
scanned or alternatively, the system can automatically 
locate this area. The operator then commands the system 

5 to begin automated scanning of the slides through a 
graphical user interface. Unattended scanning begins 
with the automatic loading of the first carrier and slide 
onto the motorized X-Y stage 38 . A bar code label 
affixed to the slide is read by a bar code reader 33 

10 during this loading operation. Each slide is then 

scanned at a user selected low microscope magnification, 
for example, lOx, to build a histological reconstruction 
or identify candidate objects based on their color, size 
and shape characteristics. The X-Y locations of 

15 candidate objects or areas of interest are stored until 
scanning is completed. 

After the low magnification scanning is completed, 
the apparatus may automatically return to each candidate 
object or area of interest, if necessary, reimaging and 

20 refocusing at a higher magnification such as 40x and 
performs further analysis to confirm the biological 
candidate. The apparatus stores an image of the object 
or area of interest for later review by a pathologist. 
All results and images can be stored to a storage device 
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21 such as a removable hard drive or optical disc or DAT 
tape or transmitted to a remote site for review or 
storage. The stored images for each slide can be viewed 
in a mosaic of images for further review. In addition, 
5 the pathologist or operator can also directly view a 
detected object or area of interest through the 
microscope using the included oculars 20 or on image 
monitor 27. 

The two system processors 102 further control an 
10 illumination controller 106 for control of substage 

illumination 48. The light output from, for example, a 
halogen light bulb, which supplies illumination for the 
system, can vary over time due to bulb aging, changes in 
optical alignment, and other factors. In addition, 
15 slides which have been "over stained" can reduce the 

camera exposure to an unacceptable level. To compensate 
for these effects, the illumination controller 106 is 
included. This controller is used in conjunction with 
light control software to compensate for the variations 
20 in light level. The light control software samples the 
output from the camera at intervals (such as between 
loading of slide carriers), and commands the controller 
to adjust the light level to the desired levels. In this 
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way, light control is automatic and transparent to the 
user and adds no additional time to system operation. 

The system processors 23 are preferably comprised of 
dual parallel Intel Pentium Pro 400 MHZ devices. The 
5 image processor 25 is preferably a Matrox Genesis board. 
The computer, in a preferred embodiment, operates under 
Windows NT. It will be recognized that any number of 
processors and operating systems can be used in the 
methods and in conjunction with the present invention. 
10 Referring now to FIG. 3 and 4, further detail of the 

apparatus 10 is shown. FIG. 3 shows a plan view of the 
apparatus 10 with the housing 12 removed. A portion of 
the automatic slide feed mechanism 37 is shown to the 
left of the microscope subsystem 32 and includes slide 
15 carrier unloading assembly 34 and unloading platform 36 
which in conjunction with slide carrier unloading hopper 
18 function to receive slide carriers which have been 
analyzed. 

Vibration isolation mounts 40, shown in further 
20 detail in FIG. 4, are provided to isolate the microscope 
subsystem 32 from mechanical shock and vibration that can 
occur in a typical laboratory environment. In addition 
to external sources of vibration, the high speed 
operation of the X-Y stage 38 can induce vibration into 
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the microscope subsystem 32. Such sources of vibration 
can be isolated from the electro-optical subsystems to 
avoid any undesirable effects on image quality. The 
isolation mounts 40 comprise a spring 40a and piston 40b 

5 submerged in a high viscosity silicon gel which is 

enclosed in an elastomer membrane bonded to a casing to 
achieve damping factors on the order of 17% to 20%. 

The automated slide handling subsystem operates on a 
single slide carrier at a time. A slide carrier 60 is 

10 shown in FIG. 5a & 5b, which provide a top view and a 
bottom view respectively. The slide carrier 60 includes 
up to four slides 70 mounted with adhesive tape 62. The 
carrier 60 includes ears 64 for hanging the carrier in 
the output hopper 18. An undercut 66 and pitch rack 68 

15 are formed at the top edge of the slide carrier 60 for 
mechanical handling of the slide carrier. A keyway 
cutout 65 is formed in one side of the carrier 60 to 
facilitate carrier alignment. A prepared slide 72 
mounted on the slide carrier 60 includes a sample area 

20 72a and a bar code label area 72b. 

FIG. 6a provides a top view of the slide handling 
subsystem which comprises a slide input module 15, a 
slide output module 17 and X-Y stage drive belt 50. FIG. 
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6b provides a partial cross-sectional view taken along 
line A-A of FIG. 6a. 

The slide input module 15 includes a slide carrier 
input hopper 16, loading platform 52 and slide carrier 

5 loading subassembly 54. The input hopper 16 receives a 
series of slide carriers 60 (FIG. 5a and 5b) in a stack 
on loading platform 52. A guide key 57 protrudes from a 
side of the input hopper 16 to which the keyway cutout 65 
(FIG. 5a) of the carrier is fit to achieve proper 

10 alignment. 

The input module 15 further includes a revolving 
indexing cam 56 and a switch 90 mounted in the loading 
platform 52, the operation of which is described further 
below. The carrier subassembly 54 comprises an infeed 

15 drive belt 59 driven by a motor 86. The infeed drive 
belt 59 includes a pusher tab 58 for pushing the slide 
carrier horizontally toward the X-Y stage 38 when the 
belt is driven. A homing switch 95 senses the pusher tab 
58 during a revolution of the belt 59. 

20 Referring specifically to FIG. 6a, the X-Y stage 38 

is shown with x position and y position motors 96 and 97 
respectively which are controlled by the microscope 
controller 31 (FIG. 9) and are not considered part of the 
slide handling subsystem. The X-Y stage 38 further 
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includes an aperture 55 for allowing illumination to 
reach the slide carrier. A switch 91 is mounted adjacent 
the aperture 55 for sensing contact with the carrier and 
thereupon activating a motor 87 to drive stage drive belt 
5 50 (FIG. 6b} . The drive belt 50 is a double-sided timing 
belt having teeth for engaging pitch rack 68 of the 
carrier 60 {FIG. 5b) . 

The slide output module 17 includes slide carrier 
output hopper 18, unloading platform 6, and slide carrier 
10 unloading subassembly 34. The unloading subassembly 34 
is a motor 8 9 for rotating the unloading platform 36 
about shaft 98 during an unloading operation described 
further below. An outfeed gear 93 driven by motor 88 
rotatably engages the pitch rack 68 of the carrier 60 
15 (FIG. 5b) to transport the carrier to a rest position 
against switch 92 . A spring loaded hold-down mechanism 
holds the carrier in place on the unloading platform 36. 

The slide handling operation is now described. 
Referring to FIG. 7, a series of slide carriers 60 are 
20 shown stacked in input hopper 16 with the top edges 60a 
aligned. As the slide handling operation begins, the 
indexing cam 56 driven by motor 85 advances one 
revolution to allow only one slide carrier to drop to the 
bottom of the hopper 16 and onto the loading platform 52. 
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FIG. 7a-7d show the cam action in more detail. The 
indexing cam 56 includes a hub 56a to which are mounted 
upper and lower leaves 56b and 5 6c respectively. The 
leaves 56b & 56c are semicircular projections oppositely 
5 positioned and spaced apart vertically. In a first 
position shown in FIG. 8a, the upper leaf 56b supports 
the bottom carrier at the undercut portion 66. At a 
position of the indexing cam 56 rotated 180°, shown in 
FIG. 7b, the upper leaf 56b no longer supports the 

10 carrier and instead the carrier has dropped slightly and 
is supported by the lower leaf 56c. FIG. 8c shows the 
position of the cam 56 rotated 270° wherein the upper 
leaf 56b has rotated sufficiently to begin to engage the 
undercut 66 of the next slide carrier while the opposite 

15 facing lower leaf 56c still supports the bottom carrier. 
After a full rotation of 360° as shown in FIG. 7d, the 
lower leaf 5 6c has rotated opposite the carrier stack and 
no longer supports the bottom carrier which now rests on 
the loading platform 52. At the same position, the upper 

20 leaf 56b supports the next carrier for repeating the 
cycle. 

Referring again to FIG. 6a & 6b, when the carrier 
drops to the loading platform 52, the contact closes 
switch 90 which activates motors 8 6 and 87. Motor 8 6 
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drives the infeed drive belt 59 until the pusher tab 58 
makes contact with the carrier and pushes the carrier 
onto the X-Y stage drive belt 50. The stage drive belt 
50 advances the carrier until contact is made with switch 
5 91, the closing of which begins the slide scanning 

process described further herein. Upon completion of the 
scanning process, the X-Y stage 38 moves to an unload 
position and motors 8, and 88 are activated to transport 
the carrier to the unloading platform 36 using stage 
10 drive belt 50. Motor 88 drives outfeed gear 93 to engage 
the carrier pitch rack 68 of the carrier 60 (FIG. 5b) 
until switch 92 is contacted. Closing switch 92 
activates motor 89 to rotate the unloading platform 36. 
The unloading operation is shown in more detail in 
15 end views of the output module 17 (FIG. 7a-7d) . In FIG. 
7a, the unloading platform 36 is shown in a horizontal 
position supporting a slide carrier 60. The hold-down 
mechanism 94 secures the carrier 60 at one end. FIG. 7b 
shows the output module 17 after motor 89 has rotated the 
20 unloading platform 36 to a vertical position, at which 
point the spring loaded hold-down mechanism 94 releases 
the slide carrier 60 into the output hopper 18. The 
carrier 60 is supported in the output hopper 18 by means 
of ears 64 (FIG. 5a and 5b). FIG. 7c shows the unloading 
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platform 16 being rotated back towards the horizontal 
position. The platform 36 rotates upward and contacts 
the deposited carrier 60. The upward movement pushes the 
carrier toward the front of the output hopper 18. FIG. 

5 7d shows the unloading platform 36 at its original 

horizontal position after having output a series of slide 
carriers 60 to the output hopper 18. 

The aspects of the apparatus 10 relating to scanning, 
focusing and image processing are further described in 

10 Patent Application 08/758,436 which is incorporated 
herein. 



Computer Implementation 

Aspects of the invention may be implemented in 

15 hardware or software, or a combination of both. However, 
preferably, the algorithms and processes of the invention 
are implemented in one or more computer programs 
executing on programmable computers each comprising at 
least one processor, at least one data storage system 

20 (including volatile and non-volatile memory and/or 
storage elements), at least one input device, and at 
least one output device. Program code is applied to input 
data to perform the functions described herein and 
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generate output information. The output information is 
applied to one or more output devices, in known fashion. 

Each program may be implemented in any desired 
computer language (including machine, assembly, high 
5 level procedural, or object oriented programming 

languages) to communicate with a computer system. In any 
case, the language may be a compiled or interpreted 
language. 

Each such computer program is preferably stored on a 
10 storage media or device (e.g., ROM, CD-ROM, tape, or 
magnetic diskette} readable by a general or special 
purpose programmable computer, for configuring and 
operating the computer when the storage media or device 
is read by the computer to perform the procedures 
15 described herein. The inventive system may also be 
considered to be implemented as a computer-readable 
storage medium, configured with a computer program, where 
the storage medium so configured causes a computer to 
operate in a specific and predefined manner to perform 
20 the functions described herein. 

A number of embodiments of the present invention have 
been described. Nevertheless, various modifications may 
be made without departing from the spirit and scope of 
the invention. Accordingly, the invention is not to be 
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limited by the specific illustrated embodiment, but only 
by the scope of the appended claims. 
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1. A method for automated image analysis of a biological 
5 specimen, comprising: 

(a) providing a biological sample to be analyzed; 

(b) automatically scanning the sample at a plurality 
of coordinates; 

(c) automatically obtaining an image at each of said 
10 coordinates; 

(d) automatically reconstructing an image of the 
sample from each individual image to create a 
reconstructed image of the sample; and 

(e) processing the reconstructed image to identify a 
15 candidate object or area of interest. 



2. The method of claim 1, further comprising: 

(a) automatically identifying a coordinate of the 
candidate object or area of interest in the reconstructed 

20 image; and 

(b) automatically acquiring an image of the object or 
area of interest, at the location coordinates obtained 
from the reconstructed image. 
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3. The method of claim 1, wherein the object or area of 
interest is detected by immunohistochemistry . 



4. The method of claim 1, wherein the object or area of 
5 interest is detected by in situ hybridization. 

5. The method of claim 1, wherein the object or area of 
interest is detected by a stain. 

10 6. The method of claim 5, wherein the stain is a nucleic 
acid dye selected from the group consisting of 
hematoxylin, Giemsa stain, methyl green, Nuclear 
Fast_Red, Hoechst 33342, Hoechst 33258, thiazole orange, 
DAPI, ethidium bromide, propidium iodide, TOTO, YOYO-1, 

15 SYTOX Blue, SYTOX Green, 7-Aminoactinomycin, 9-Amino-6- 
chloro-2-methoxyacridine, and acridine homodimer. 

7. The method of claim 5, wherein the object or area of 
interest is stained with a cytoplasmic dye such as eosin 

20 or Kleihauer_Betke cytochemical stain or a combination 
thereof. 

8. The method of claim 1, wherein the object or area of 
interest is a cell specific marker. 
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9. The method of claim 8, wherein the cell specific 
marker is detected by a nuclear stain and counterstain. 

5 10. The method of claim 8, wherein the cell specific 
marker is detected by immunohistochemistry, in situ 
hybridization, staining or a combination thereof. 

11. The method of claim 1, wherein the reconstructed 
10 image is a digital image. 

12. A method for histological reconstruction, comprising: 
{a) providing a sample of a biological specimen on a 

series of slides, wherein consecutive samples are: 
15 (1) stained by a nuclear stain such as 

hematoxylin and eosin (H/E) ; or 

(2) detectably labeled by an 
immunohistochemistry (IHC) method and counterstained; or 

(3) detectably labeled by an in situ 

20 hybridization (ISH) method and counterstained; or 

(4) detectably labeled by a combined (IHC) and 
(ISH) method and counterstained; 

(b) automatically obtaining , an image from each of 
the samples 
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(c) automatically reconstructing the sample images 
such that a first sample image is paired with a 
consecutive sample having a corresponding image; and 

(d) automatically performing a histological 
5 reconstruction of the specimen. 

13. The method of claim 12, wherein the first sample and 
consecutive sample have overlapping images. 

10 14. A computer program, residing on a computer_readable 
medium, for automated image analysis of biological 
specimen, the computer program comprising instructions 
for causing a computer to: 

a) process a sample, wherein the sample is stained 

15 with a nucleic acid dye and a counterstain; 

(b) scan the sample at a plurality of coordinates; 

(c) obtain an image at each of said coordinates; 

(d) reconstruct the sample from the individual 
images to create a reconstructed image of the sample; 

20 (e) identify a coordinate of a candidate object or 

area of interest in the reconstructed sample; and 

(f) acquire an image of the object of interest, at 
the location coordinates obtained from the reconstructed 
sample . 
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15. The computer program of claim 14, wherein the 
reconstructed image of the sample is a digital image. 

5 16. A computer program, residing on a computer_readable 
medium, for histological reconstruction, the computer 
program comprising instructions for causing a computer 
to: 

(a) generate, digitize, and store in memory images 
10 from a biological specimen on a series of slides, wherein 
consecutive samples are: 

(1) stained by a nuclear stain such as 
hematoxylin and with eosin; or 

(2) detect ably labeled by an immunohistochemical 
15 (IHC) method and counters tained; or 

(3) detectably labeled by an in situ 
hybridization {ISH) method and counterstained; or 

(4) detectably labeled by a combined IHC and ISH 
method and counterstained; 

20 (b) place the stored sample images in an order such 

that a nucleic acid stained and counterstained sample is 
paired with a consecutive IHC or ISH sample; and 

(c) automatically perform a histological 
reconstruction of the specimen. 
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